SHEFT: An Elastic Workflow Scheduling Algorithm for Cloud Computing

Due to the complexity of scientific processes, compute and data intensive scientific workflows are often required to be executed in a distributed high-end computing environment, such as the Cloud. Most existing workflow scheduling algorithms only consider a computing environment in which the number of compute resources is bounded. Resources assigned to a workflow cannot be automatically determined on demand of the size of the workflow and they are not released to the environment until an execution of the workflow completes. The salient features of Cloud that provide an illusion of unlimited resources to requesters and allow resources to scale on demand as usage changes through dynamic provisioning bring a new opportunity to solve this scheduling problem. To address the problem, we firstly formalize a model of a Cloud computing environment and a workflow graph representation for the environment. Then, we propose SHEFT, a workflow scheduling algorithm that are able to schedule workflows given the elastically changing compute resources. Our extensive experiments and comparisons for large-scale, compute-intensive, and data-intensive workflows has shown that SHEFT not only outperforms several representative workflow scheduling algorithms in optimizing workflow execution time, but also enables resources to scale elastically during workflow execution. Keywords-Workflow scheduling; elastic scaling; cloud computing; heterogeneous environment.